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Foreword 


Navy  surveys  and  polls  yield  valuable  information  that  is  used  to  evaluate  programs 
and  improve  policy.  In  the  past,  mail-out  surveys  have  been  the  norm  in  the  Navy,  but 
they  are  costly  and  time-consuming.  To  alleviate  these  issues,  a  Quick  Poll  methodology 
was  created  to  reduce  costs  and  turnaround  time.  Because  this  is  a  new  methodology,  its 
efficiency  and  effectiveness  need  to  be  evaluated.  This  report  details  findings  from  one 
process  improvement  study  undertaken,  specifically  looking  at  the  issue  of  non¬ 
response  on  surveys  and  polls. 

The  authors  would  like  to  thank  the  Command  Career  Counselors,  other  command 
points-of-contact,  and  the  Sailors  who  complete  the  polls  for  making  the  Quick  Poll 
methodology  possible.  The  authors  also  acknowledge  Ms.  Evangeline  Clewis  for  her 
expert  assistance  with  the  follow-up  survey. 
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Executive  Summary 


Problem 

The  Quick  Poll  methodology  is  relatively  new  to  the  Navy,  and  needs  to  be  studied  to 
determine  where  improvements  are  required.  A  key  aspect  to  be  studied  is  whether 
respondents  to  a  Quick  Poll  differ  from  non-respondents. 

Objective 

The  current  study  had  two  main  objectives:  (l)  determine  if  there  are  differences 
between  responders  and  non-responders,  and  (2)  determine  if  levels  of 
conscientiousness  are  related  to  whether  or  not  a  person  responds  to  a  survey. 

Approach 

A  short  follow-up  survey  was  sent  to  5,580  Sailors  who  had  been  asked  previously  to 
take  a  Quick  Poll.  The  survey  included  questions  from  the  original  Quick  Poll  as  well  as 
questions  about  the  Quick  Poll  methodology  and  questions  to  assess  a  person’s 
conscientiousness.  Different  colored  business  reply  envelopes  were  used  to  differentiate 
those  who  had  responded  to  the  original  survey  from  those  who  had  not. 

Findings 

1.  Overall,  there  were  no  statistically  significant  differences  between  responders  and 
non-responders  to  the  original  Quick  Poll. 

2.  Rates  of  correct  answers  to  knowledge  questions  were  higher  on  the  follow-up 
survey  than  on  the  original  Quick  Poll. 

3.  Conscientiousness  scale  scores  were  not  statistically  different  between 
responders  and  non-responders. 

Recommendations 

1.  Determine  more  efficient  means  of  contacting  people  quickly. 

2.  Conduct  additional  Quick  Poll  follow-up  studies  to  continue  refining  the 
methodology. 
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I  ntroduction 


Background 

Quick  Polls  were  developed  to  obtain  valid  and  reliable  attitudes  and  opinions  of  U.S. 
Navy  personnel  in  a  shorter  time  period  than  traditional  mailout  surveys.  While 
traditional  Navy  surveys  have  typically  been  conducted  in  six  months  to  one  year,  Quick 
Polls  are  completed  in  13-21  business  days,  from  question  design  to  results 
presentation.  In  the  Quick  Poll  process,  short,  targeted  surveys  are  administered 
entirely  on  the  Internet  to  stratified  random  samples  of  Navy  Sailors.  Navy  commands 
are  notified  of  respondents  via  the  Navy’s  internal  messaging  system;  the  message 
requests  that  individuals  who  are  listed  in  the  message  be  notified  that  they  have  been 
selected  to  take  part  in  a  Quick  Poll  and  that  they  go  to  the  Internet  site  listed  to 
complete  it.  The  assistance  of  command  leadership  is  requested  in  notifying  personnel 
selected  to  take  the  poll,  but  completion  of  the  poll  is  voluntary.  Because  Navy 
Command  Career  Counselors  (CCCs)  interact  with  such  a  large  proportion  of  command 
personnel,  the  message  recommends  using  CCCs  to  contact  individuals. 

Similar  to  most  large-scale,  mailout  Navy  and  Department  of  Defense  surveys,  Quick 
Polls  typically  yield  a  30  percent  response  rate,  although  this  is  obtained  in  a  matter  of 
days  rather  than  the  months  needed  on  large  scale  surveys.  Previous  research  has 
investigated  survey  non-response  on  traditional  mailout  Navy  surveys  (Newell, 
Rosenfeld,  Harris,  &  Hindelang,  2004).  In  that  study,  individuals  who  were  selected  for 
the  1999  Navy  Equal  Opportunity/Sexual  Harassment  Survey  were  resurveyed  over  a 
year  later  and  asked  to  indicate  whether  they  had  responded  or  not  responded  to  the 
original  survey.  It  was  found  that  self-reported  responders  and  non-responders  did  not 
differ  on  attitudinal  items,  but  self-reported  responders  did  tend  to  be  more  likely  to 
report  attending  various  Navy  training  than  did  non-responders.  This  suggests  that 
responders  are  more  compliant  or  conscientious  than  non-responders  are. 

While  we  have  some  information  on  survey  non-responders  for  surveys  using  more 
traditional  mail-out  methods,  the  different  survey  contact  methods  used  with  Quick 
Polls  warrant  the  investigation  of  non-response  issues  on  these  polls  as  well.  The  goals 
of  this  study  are  to  determine  the  reasons  why  respondents  did  not  complete  Quick  Polls 
and  to  investigate  if  non-responders  to  Quick  Polls  differ  from  responders  in  their 
answers  to  Quick  Poll  questions. 

I  ndividual  Difference  Factors  and  Survey  Response 

In  the  study  examining  Navy  non-response  mentioned  above,  Newell  et  al.  (2004) 
found  an  interesting  difference  when  analyzing  results  by  self-reported  responders  and 
non-responders— self-reported  responders  were  more  likely  to  attend  Navy  training 
than  non-responders.  Assuming  that  the  similarities  on  the  attitudinal  items  accurately 
reflect  a  lack  of  difference  between  responders  and  non-responders  on  key  survey  items, 
this  finding  of  differences  on  these  factual  training  attendance  items  suggests  that 
individual  differences,  such  as  personality  traits  (e.g.,  compliance,  conscientiousness)  or 
organizational  commitment,  could  impact  the  tendency  to  respond  to  surveys.  It  may  be 


1 


that  those  who  are  more  committed  to  an  organization,  or  are  more  conscientious  in 
general,  are  more  likely  to  complete  surveys  and  comply  with  organizational  functions 
such  as  training.  Response  rate  may  be  more  a  function  of  some  respondents  simply 
being  good  organizational  citizens,  rather  than  those  not  responding  possessing  more 
negative  attitudes  about  the  Navy. 

Several  studies  have  investigated  the  relationship  between  survey  response  and 
individual  difference  factors  (Gershen  &  McCreary,  1983;  Helgeson,  Voss,  &  Terpening, 
2002;  Johnson  &  Mowrer,  2000;  Marcus  &  Schutz,  2005;  Rogelberg,  Conway, 
Sederberg,  Spitzmuller,  Aziz,  &  Knight,  2003).  However,  these  studies  have  found 
mixed  results  between  various  personality  variables  and  survey  response.  The  individual 
difference  measures  that  appear  to  have  the  best  potential  link  to  survey  responsiveness 
include  agreeableness  and  conscientiousness  (the  tendency  to  be  responsible, 
dependable,  persistent,  and  achievement  oriented  [Digman,  1990]).  Gershen  and 
McCreary  (1983)  and  Johnson  and  Mowrer  (2000)  found  no  significant  associations 
between  response  rate  and  individual  differences.  However,  Marcus  and  Schutz  (2005) 
did  find  differences  between  responders  and  non-responders  on  agreeableness,  but  did 
not  find  differences  between  these  two  groups  on  conscientiousness.  Further,  Rogelberg, 
Conway,  Sederburg,  Spitzmuller,  Aziz,  and  Knight  (2003)  investigated  differences 
between  responders  and  non-responders  on  the  personality  variables  of  agreeableness 
and  conscientiousness,  as  well  as  other  relevant  organizational  measures.  They  found 
that  active  non-responders  (i.e.,  those  who  indicated  that  they  would  not  complete  a 
survey)  differed  from  responders  on  satisfaction,  agreeableness,  and  conscientiousness. 
However,  passive  non-responders  (those  who  indicated  that  they  would  complete  a 
survey  and  subsequently  did  not  complete  the  follow-up)  did  not  differ  from  responders 
on  these  variables. 

The  purposes  of  this  study  are  twofold.  The  first  is  to  provide  a  direct  test  of  potential 
differences  between  responders  and  non-responders  on  an  operational  Navy  Quick  Poll. 
Non-response  follow-ups  in  military  surveys  have  been  rare,  but  with  lower  survey 
response  rates  there  is  a  need  to  demonstrate  respondent/non-respondent  equivalency 
to  support  the  validity  of  the  findings.  Second,  the  study  examines  whether 
conscientiousness  is  associated  with  the  tendency  to  respond  to  Quick  Polls. 


Method 


A  short  follow-up  survey  was  administered  to  a  group  of  U.S.  Navy  officers  and 
enlisted  personnel  (AT  =  5,580)  who  were  previously  asked  to  complete  a  Navy  Quick 
Poll.  The  original  Quick  Poll  was  Internet-based  and  contained  22  closed-ended  items 
and  one  open-ended  item  designed  to  evaluate  the  impact  of  a  recent  Navy  media 
campaign.  The  sample  for  the  original  poll  was  selected  using  a  two-stage  cluster, 
stratified  random  sample  approach  (Newell,  Dever,  &  Rosenfeld,  2004)  with  the  results 
statistically  weighted  to  allow  generalizing  to  the  larger  Navy  population. 
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The  follow-up  survey  was  a  l-page,  paper-based  survey  sent  to  all  individuals  in  the 
original  sample.  To  identify  responders  and  non-responders  to  the  original  poll,  the 
follow-up  survey  reply  envelopes  were  color-coded  to  distinguish  original  poll 
respondents  from  non-respondents,  based  upon  the  login  information  of  the  original 
poll.  The  follow-up  survey  included  questions  regarding  participation  and  completion  of 
the  initial  poll,  demographics  (i.e.,  military  paygrade),  several  questions  from  the 
original  poll,  and  a  brief  conscientiousness  inventory  taken  from  the  International 
Personality  Item  Pool  (2001).  The  follow-up  survey  questions  are  contained  in  the 
Appendix. 


Results 


A  total  of  975  Navy  personnel  completed  the  follow-up  survey  (response  rate  of 
17.4%),  of  which  41  percent  (n  =  398)  completed  the  original  Quick  Poll  and  59  percent 
(n  =  577)  did  not.  Of  the  respondents,  40  percent  were  enlisted  (n  =  388)  and  60 
percent  (n  =  572)  were  officers  (15  respondents  did  not  indicate  their  rank).  The 
analyses  were  based  on  records  of  individuals’  response  or  non-response  to  the  original 
Quick  Poll,  and  not  on  their  self-reported  participation.1 

The  participants  who  did  not  complete  the  original  Quick  Poll  were  asked  to  identify 
reasons  why  they  did  not  complete  it.  A  large  percentage  did  not  respond  to  the  initial 
Quick  Poll  questionnaire  because  they  were  not  told  to  complete  it  (53%).  Eleven 
percent  of  the  respondents  who  did  not  complete  the  first  poll  reported  technical 
problems  (e.g.,  “the  web  page  would  not  load”  or  “web  site  blocked”),  while  another  11 
percent  indicated  time  issues  (e.g.,  more  important  work  and  limited  computer  time). 
Another  30  percent  reported  “Other”  factors  contributed  to  their  non-response,  such  as 
not  knowing  the  deadline,  transitioning  between  assignments,  or  having  limited  access 
to  the  Internet  (see  Table  1). 


1  Of  those  who  returned  the  follow-up  survey,  approximately  50%  (n  =  484)  indicated  that  they  completed 
the  first  Quick  Poll.  However,  for  17%  (n  =  97)  of  those  who  reported  completing  the  first  survey  no  initial 
poll  responses  were  received.  Of  those  who  indicated  not  completing  the  initial  survey  (n  =  437),  2  percent 
(n  =  7)  reported  they  had  actually  returned  it.  Since  self-reported  recollections  may  not  be  accurate,  it  was 
decided  to  assign  respondents  to  responder  and  non-responder  groups  based  on  whether  the  records 
indicated  they  actually  responded  or  did  not. 
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Table  1 

Factors  contributing  to  Sailors'  non-response 


Reason 

Percentage  of  Non¬ 
responders 

1  was  not  told  to  complete  the  poll. 

53 

There  are  more  important  work  demands  on  my  time. 

9 

The  survey  Web  page  would  not  load. 

9 

The  poll  was  no  longer  available. 

6 

The  Web  site  was  blocked  due  to  security  reasons. 

2 

1  have  limited  time  to  use  the  computer  and  prefer  to  use  it 

2 

for  other  things. 

1  dislike  computer  surveys. 

2 

1  don't  believe  that  survey  results  are  used  to  improve 

2 

programs/policies. 

1  was  not  interested  in  the  topic. 

1 

1  did  not  want  to  do  it. 

<  1 

1  was  concerned  someone  in  my  chain  of  command  would 

<  1 

see  the  answers. 

1  don't  trust  your  organization  to  keep  my  answers 

<  1 

confidential. 

Other 

30 

Responders  to  the  original  poll  were  also  asked  who  notified  them  of  the  poll.  Data 
from  the  follow-up  study  indicated  that  most  Sailors  were  notified  by  senior  leadership 
(e.g.,  their  Commanding  Officers  [CO],  Executive  Officer  [XO],  Department  Head,  or 
Command  Career  Counselors  [CCC]).  Interestingly,  a  sizable  proportion  stated  that 
someone  else  told  them  to  complete  it  (32%).  Of  the  non-responders,  73  percent 
indicated  that  nobody  told  them  to  complete  the  survey. 
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Table  2 

Source  of  information  about  the  original  poll  by  respondent  type 


Percent 

Did  not 

Responded  to  respond  to 

1  nitial  Survey  initial  survey 

CO 

14 

3 

xo 

12 

3 

Department  Head 

10 

3 

Division  Head 

7 

1 

Command  Master  Chief/Chief  of  Boat 

6 

2 

Command  Career  Counselor 

9 

3 

Other 

32 

9 

No  One 

7 

73 

Selected  questions  from  the  original  poll  were  also  included  on  the  follow-up  survey. 
These  included  questions  regarding  respondent’s  knowledge  of  material  presented  in  a 
Navy  media  campaign.  Of  these  knowledge  questions,  the  majority  of  respondents 
answered  incorrectly  (65%  of  original  responders;  69%  of  non-responders).  In 
comparison  to  the  results  from  the  original  poll,  80  percent  of  the  respondents 
answered  incorrectly. 

Participants  were  questioned  about  the  sources  they  use  to  get  information  about 
current  Navy  operations.  The  majority  of  participants  reported  receiving  information 
from  the  Navy  Times  (57%;  a  publication  by  a  private  company)  and  official  e-mail  or 
Navy  Knowledge  Online,  an  official  Navy  website  (56%).  Several  other  important 
sources  regarding  Navy  operations  included  the  CO’s  Call  (44%),  coworkers  (46%),  news 
channels  (e.g.,  Fox  news,  CNN,  local  news)  (46%),  and  the  Internet  (40%).  There  were 
no  differences  found  in  the  sources  used  for  information  between  those  who  actually 
returned  the  Quick  Poll  and  those  who  did  not  (ps  >  .05).  These  results  are  quite  similar 
to  those  obtained  in  the  original  poll.  The  biggest  difference  is  that  coworkers  (58%)  and 
the  CO’s  Call  (56%)  were  the  most  reported  sources  of  information  in  the  original  poll. 

The  participants  were  also  asked  to  what  degree  they  agreed  or  disagreed  with  three 
statements  regarding  the  Fleet  Response  Plan  (a  newly  implemented  Navy  policy  at  the 
time  of  the  original  poll  and  the  subject  of  the  media  campaign).  A  fair  number  of 
participants  (43%  responders,  42%  non-responders)  agreed  that  they  have  a  good 
understanding  of  the  Fleet  Response  Plan  (see  Table  3).  The  chi  square  analysis  was  non 
significant  (x2  (2,  N  =  967)  =  0.52,  p  =  .77).  In  the  original  poll,  31  percent  agreed  they 
had  a  good  understanding  of  Fleet  Response  Plan. 
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Table  3 

Sailor  responses  to  the  statement  "I  have  a  good 
understanding  of  the  Fleet  Response  Plan"  by 
Respondent  Type 


Response 

Percent 

Responded 

Did  not  respond 

Agree 

43 

42 

Neutral 

23 

25 

Disagree 

35 

34 

Forty  percent  of  the  respondents  (41%  responders,  40%  non-responders)  agreed 
with  the  statement  that  “Implementation  of  the  Navy’s  Fleet  Response  Plan  will  improve 
the  Navy’s  flexibility  and  readiness”  (see  Table  4).  Again,  the  chi  square  analysis  was 
non  significant  (x2  (2,  N  =  965)  =  1.23,  p  =  .54).  In  comparison,  36  percent  of 
respondents  in  the  original  poll  agreed  with  this  statement. 


Table  4 

Sailor  responses  to  the  statement  "I  mplementation  of  the 
Navy's  Fleet  Response  Plan  will  improve  the  Navy's 
flexibility  and  readiness"  by  respondent  type 


Response 

Percent 

Responded 

Did  not  respond 

Agree 

41 

40 

Neutral 

51 

49 

Disagree 

8 

11 

One-fourth  of  participants  (27%  responders  and  25%  non-responders)  reported 
agreement  with  the  statement  “Summer  Pulse  adequately  demonstrated  the  Navy’s  Fleet 
Response  Plan”  (see  Table  5).  These  results  again  produced  a  non-significant  chi  square 
(x2  (2,  N  =  952)  =  0.70  p  =  .71).  This  finding  was  very  similar  to  the  original  poll  where 
25  percent  of  respondents  also  agreed  with  this  statement. 
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Table  5 

Sailor  responses  to  the  statement  "Summer  Pulse  '04  adequately 
demonstrated  the  Navy's  Fleet  Response  Plan"  by  respondent  type 


Response 

Percent 

Responded 

Did  not  respond 

Agree 

27 

25 

Neutral 

66 

68 

Disagree 

7 

8 

Conscientiousness 

Questions  added  to  this  follow-up  survey  included  a  10-item  measure  of 
conscientiousness  taken  from  the  International  Personality  Item  Poll  (2001).  Scores 
ranged  from  1  (not  conscientious)  to  5  (highly  conscientious).  The  mean  scale  score  was 
computed,  with  the  maximum  score  being  50,  or  highly  conscientious.  An  independent 
samples  T-test  was  performed  to  test  the  mean  scores  of  those  who  actually  responded 
to  the  original  Quick  Poll  and  those  who  did  not.  No  statistically  significant  difference 
was  found  between  those  who  responded  (M  =  42.91)  and  those  who  did  not  (M  = 
43.26),  f(i,  923)=  -.936,  p  >  .350. 

Open-ended  Comments 

Along  with  the  closed-item  questions,  respondents  were  given  the  opportunity  to 
provide  additional  comments  at  the  end  of  the  follow-up  survey.  A  total  of  369 
comments  were  provided,  and  a  short  content  analysis  of  these  responses  was 
conducted  by  NPRST  researchers.  Results  indicate  that  most  comments  contained 
statements  concerning:  (1)  time  allotted  to  complete  the  survey,  (2)  information 
regarding  knowledge  of  the  survey  and/or  survey  topic,  (3)  survey  population,  (4) 
incentives,  (5)  feedback,  (6)  survey  method  and  administration,  and/or  (7)  information 
about  the  actual  survey. 

The  largest  percentage  of  respondents  who  provided  comments  (27%)  offered 
feedback  regarding  the  survey  methodology.  Sailors  offered  ideas  for  improving  the  way 
the  surveys  are  administered,  such  as  making  the  survey  via  e-mail. 

Almost  a  fourth  of  the  comments  (22%)  indicated  a  need  for  better  communication. 
Sailors  indicated  communication  concerns;  some  did  not  know  about  the  survey 
whereas  others  did  not  have  full  knowledge  or  understanding  of  the  survey  topics.  Other 
comments  included  statements  such  as  “Get  information  out  that  there’s  a  survey  out  to 
those  that  you  want  to  poll,”  “Make  sure  we  know  about  them,”  and  “I  had  no  knowledge 
of  the  survey.”  Suggestions  for  improvement  include  advertising  the  survey  through  the 
Navy  Times,  on  Navy  Knowledge  Online  (NKO),  or  even  through  Navy  educational 
classes. 
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Feedback  and  incentives  also  comprised  a  large  percentage  of  the  open-ended 
responses  (17%).  Sailors  reported  that  they  would  be  more  interested  in  and  more  likely 
to  complete  the  surveys  if  they  were  given  the  results.  Their  comments  suggest  that  they 
want  to  know  how  completing  the  survey  will  benefit  or  affect  them  and  see  that  the 
results  are  being  taken  seriously  and  put  to  use.  They  recommended  that  results  be 
published  or  provided  via  the  Internet.  Along  with  feedback,  numerous  comments  were 
provided  asking  for  monetary  incentives  or  some  type  of  reward  system  for  completing 
the  surveys.  It  should  be  noted  that  military  policy  limits  the  types  of  incentives  that  can 
be  given  to  those  completing  surveys  while  at  work  since  participants  are  essentially 
already  being  compensated. 

Other  comments  provided  by  the  respondents  dealt  with  time  allotted  to  complete 
the  survey  (5%),  survey  population  (6%),  and  the  survey  itself  (15%).  Many  of  the 
responses  that  fell  into  one  or  more  of  these  categories  reported  needing  more  time  to 
complete  the  survey.  Some  of  the  Sailors  stated  that  they  need  more  time  due  to  slow 
mail  delivery  and  limited  Internet  access.  They  also  specified  the  need  for  timely 
notification  to  complete  the  survey.  Other  respondents  focused  their  comments  on  the 
survey  itself,  stating  the  need  for  shorter  surveys  that  included  only  the  important  Navy 
issues,  and  providing  survey  items  that  are  clearer. 


Discussion 


Overall,  there  were  no  statistically  significant  differences  between  responders  and 
non-responders  to  a  Navy  Quick  Poll,  based  on  a  follow-up  survey  that  both  groups 
completed.  Specifically,  differences  in  response  behaviors  are  not  related  to  levels  of 
conscientiousness.  This  is  contrary  to  the  findings  of  Newell,  Rosenfeld,  Harris,  and 
Hindelang  (2004),  who  found  that  conscientiousness  was  related  to  survey  response 
behaviors.  Since  respondents  to  the  Newell  et  al  (2004)  study  self-identified  as  being  a 
respondent  or  a  non-respondent,  it  is  possible  that  their  conscientiousness  finding  was 
related  to  whether  or  not  a  respondent  remembered  replying  to  a  survey  rather  than 
their  actual  behavior. 

Thus,  in  an  experimental  design  that  accurately  identified  actual  survey  responders 
and  non-responders,  the  responses  of  these  two  groups  did  not  systematically  differ. 
This  provides  a  stronger  demonstration  of  the  equivalence  of  respondents  and  non¬ 
respondents  on  Navy  surveys  and  polls  than  in  Newell,  Rosenfeld,  Harris,  and 
Hindelang  (2004)  where  respondents  and  non-respondents  were  self-identified,  rather 
than  being  identified  based  on  whether  a  database  indicated  that  they  had  actually 
responded  or  not. 

While  responders  and  non-responders  to  the  follow-up  survey  did  not  differ,  there 
was  a  larger  difference  between  the  responses  to  the  original  poll  and  responses  to  the 
follow  up  survey  on  the  knowledge  question.  For  example,  a  smaller  percentage 
answered  the  knowledge  question  incorrectly  on  the  follow-up  poll  (67%)  compared  to 
the  80  percent  of  respondents  who  answered  incorrectly  on  the  first  poll.  There  are,  in 
this  research  design,  several  factors  which  may  have  influenced  this  difference— the  2- 
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week  time  delay  between  the  original  poll  and  the  follow-up,  the  different  form  of 
administration  (Internet  versus  paper  survey)— but  the  differences  between  the 
responses  warrant  further  investigation. 

Although  the  survey  did  not  detect  any  meaningful  differences  between  responders 
and  non-responders,  the  survey  did  provide  much  useful  information  that  can  be  used 
to  improve  future  Navy  survey  response  rates.  Responses  to  both  the  closed-  and  open- 
ended  items  indicated  the  need  for  a  more  effective  and  efficient  means  of 
communication  to  improve  methods  for  administering  surveys  in  general  and  Quick 
Polls  in  particular.  In  the  U.S.  Navy,  information  often  cascades  from  higher-ranking 
personnel  to  command  leaders,  who  then  report  it  to  their  subordinates.  A  majority  of 
respondents  indicated  that  they  did  not  respond  to  the  initial  survey  because  they  were 
not  told  to  do  so,  showing  that  there  is  a  need  for  better  and  direct  contact  and 
communication  with  Sailors  as  a  way  of  increasing  survey  response  rates.  Although 
there  is  currently  no  way  to  directly  contact  personnel,  other  methods  of  contact  are 
being  explored.  Some  open-ended  responses  suggested  the  need  for  personal  contact, 
better  means  to  “get  the  word  out,”  reminders,  and  better  survey  advertising. 
Implementing  these  suggestions  might  result  in  increased  rates  of  response  on  future 
surveys  and  Quick  Polls. 

Survey  value  and  survey  enjoyment  may  also  be  important  factor  in  participation. 
Survey  value  refers  to  the  degree  to  which  respondents  perceive  the  survey  as  being 
worthwhile  (Rogelberg,  Fisher,  Maynard,  Hakel,  &  Horvath,  2001).  Survey  enjoyment 
refers  to  the  degree  to  which  the  respondent  enjoys  completing  surveys.  Participants’ 
open-ended  comments  indicated  possible  low  degrees  of  survey  value  and  survey 
enjoyment.  Comments  suggested  that  respondents  are  overwhelmed  with  surveys  on  top 
of  their  regular  duties  and  that  surveys  should  be  used  for  only  critical  Navy  issues. 
Sailors  reported  that  they  do  not  feel  this  survey  was  essential  and  that  they  have  more 
important  work  to  complete,  thus  it  is  perceived  as  having  limited  value.  Participants 
also  expressed  concern  over  a  lack  of  feedback  and  action  taken  from  the  survey  results. 
Feedback  and  action  are  critical  to  the  perception  of  survey  value  and  should  be 
included  as  a  part  of  the  survey  process.  If  participants  do  not  feel  that  their 
participation  is  worthwhile,  then  they  will  be  less  inclined  to  complete  future  surveys. 

In  sum,  this  study  adds  to  a  growing  literature  supporting  the  notion  that  the 
responses  of  survey  respondents  and  non-respondents  do  not  systematically  differ. 
Despite  the  often  reflexive  use  of  response  rates  as  an  index  of  survey  quality,  it  appears 
that  surveys  with  lower  response  rates  can  provide  scientifically  reliable  population 
estimates  that  are  not  systematically  affected  by  non-response  bias. 

Limitations 

While  the  current  study  adds  to  the  literature  on  non-response,  there  are  some 
limitations  to  the  present  study.  The  first  limitation  was  the  possible  incorrect 
categorization  of  up  to  83  participants.  It  was  determined  during  the  first  survey  that 
technical  difficulties  may  have  incorrectly  recorded  use  of  usernames  and  passwords. 
When  comparing  the  number  of  survey  responses  to  the  number  of  unique  usernames 
and  passwords,  it  was  discovered  that  there  were  83  more  survey  responses  than  used 
usernames  and  passwords.  It  is  likely  that  individuals  entered  the  survey,  completed  the 
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survey,  but  their  usernames  and  passwords  were  never  deactivated  after  their  responses 
were  submitted.  This  is  likely  given  the  feedback  from  a  small  number  of  Sailors  who 
called  to  report  that  after  they  pressed  the  “submit”  button  on  the  original  survey,  they 
received  a  “timed  out”  response  and  were  concerned  that  their  responses  did  not  post. 
As  username  was  not  linked  to  actual  survey  responses  for  this  Quick  Poll,  it  is 
impossible  to  determine  if  their  responses  were  posted  to  the  database.  Therefore,  there 
is  the  possibility  that  some  individuals  did  complete  the  survey,  but  were  classified  as 
“non-responders”  according  to  our  records. 

To  assess  the  impact  of  this  scenario,  additional  analyses  were  conducted.  When 
comparing  the  results  of  the  analyses  using  participants’  report  of  participation  versus 
the  researchers  record  of  participation,  it  appeared  that  there  were  no  significant 
differences  in  the  output.  Therefore,  impact  of  this  potential  incorrect  classification  of 
respondents  appears  to  be  minimal.  This  issue,  although  insignificant  in  this  study, 
could  pose  a  problem  in  future  studies  and  must  be  dealt  with  to  assure  individuals  are 
correctly  identified  as  responders  and  non-responders. 

Other  possible  limitations  are  related  to  aspects  of  this  research  design.  For  example, 
in  the  original  poll,  Sailors  were  to  be  notified  by  command  leaders.  In  the  follow-up 
survey,  members  of  the  sample  received  a  personalized,  individual  letter  at  their 
command  address.  This  personalized  letter  may  have  an  impact  on  participation  and 
influenced  the  results.  Further,  the  follow-up  survey  was  shorter  than  the  original  poll 
and  did  not  require  any  access  to  the  Internet.  These  possible  mode  effects  may  have 
impacted  the  results  as  noted  above. 

Finally,  the  response  rate  to  the  follow-up  survey  was  much  lower  than  the  response 
rate  to  the  original  poll.  While  perhaps  not  unexpected,  especially  among  individuals 
who  had  already  completed  the  original  Quick  Poll,  there  remains  the  possibility  of  self¬ 
selection,  which  could  obscure  differences  between  groups. 
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Appendix: 

Non-response  Survey 


A-o 


Dear  Sailor, 


We  are  conducting  a  study  to  learn  how  Sailors  like  you  feel  about  Navy  surveys  in  which  you  are  invited  to  participate.  An 
example  of  a  recent  survey  is  the  CNP  Navy  Quick  Poll  about  the  Summer  Pulse  ’04  Exercise.  This  survey  was  conducted 
20  -  27  October,  and  asked  about  opinions  and  knowledge  of  the  Fleet  Response  Plan  (FRP)  and  the  Summer  Pulse  ’04 
Exercise. 

This  is  a  follow-on  study  to  that  CNP  Navy  Quick  Poll,  designed  to  help  us  find  ways  to  increase  the  number  of  Sailors  who 
take  the  time  to  provide  us  their  opinions.  A  few  questions  from  the  Summer  Pulse  ’04  poll  will  be  used  as  a  "quality  check" 
to  see  if  the  responses  we  are  currently  gathering  are  similar  to  those  obtained  on  that  poll. 

If  you  have  any  questions,  please  contact  the  Navy  Survey  Line  at  (901)  874-2202,  DSN  882-2202,  or 
MILL_Navyquickpolling@navy.mil. 


PRIVACY  ACT  STATEMENT 

Authority  to  request  this  information  is  granted  under  Title  5,  U.S.  Code  301,  and  Department  of  the  Navy  Regulations. 
License  to  administer  this  survey  is  granted  under  OPNAV  Report  Control  Symbol  1 000-4,  which  expires  on  28  February 
2005. 

PURPOSE:  The  purpose  of  this  questionnaire  is  to  collect  data  to  evaluate  existing  and  proposed  Navy  personnel  policies, 
procedures,  and  programs. 

ROUTINE  USES:  The  information  provided  in  this  questionnaire  will  be  analyzed  by  the  Navy  Personnel  Research,  Studies, 
and  Technology  Department.  The  data  files  will  be  maintained  by  the  Navy  Personnel  Research,  Studies,  and  Technology 
Department  where  they  may  be  used  for  determining  changing  trends  in  the  Navy. 

ANONYMITY:  All  responses  will  be  held  in  confidence  by  the  Navy  Personnel  Research,  Studies,  and  Technology 
Department.  Information  you  provide  will  be  considered  only  when  statistically  summarized  with  the  responses  of  others,  and 
will  not  be  attributable  to  any  single  individual. 

PARTICIPATION:  Completion  of  this  questionnaire  is  entirely  voluntary.  Failure  to  respond  to  any  of  the  questions  will  NOT 
result  in  any  penalties  except  possible  lack  of  representation  of  your  views  in  the  final  results  and  outcomes. 


MARKING  INSTRUCTIONS 


•  Use  a  No.  2  pencil  only. 

•  Do  not  use  ink,  ballpoint,  or  felt  tip  pens. 

•  Make  solid  marks  that  fill  the  response  completely. 

•  Erase  cleanly  any  marks  you  wish  to  change. 

•  Make  no  stray  marks  on  this  form. 


f  MIHftJ -  ■  ICC  a  :  t 


INCORRECT:  <2f®(S>© 
CORRECT:  • 


Please  complete  this  survey  as  soon  as  possible  and  return  in  the  enclosed  envelope  to: 


Commanding  Officer 
Survey  Operations  Center  (SOC) 

Navy  Personnel  Research,  Studies,  and  Technology  (PERS-14) 

5720  Integrity  Drive 
Millington,  TN  38055-1400 


A-i 


SUMMER  PULSE  04  FOLLOW-ON  SURVEY 


1.  Did  you  complete  the  recent  web-based  CNP 
Navy  Quick  Poll  about  Summer  Pulse  '04? 

CD  Yes  (skip  to  Question  3) 

CD  No 
CD  Don’t  know 

2.  If  you  did  not  complete  the  CNP  Navy  Quick 
Poll,  why  not?  (Mark  all  that  apply.) 

CD  |  was  not  told  to  complete  the  poll 
CD  |  was  not  interested  in  the  topic 
CD  The  survey  Web  page  would  not  load 
CD  I  did  not  want  to  do  it 
CD  |  was  concerned  someone  in  my  chain  of 
command  would  see  the  answers 
CD  The  Web  site  was  blocked  due  to  security 
reasons 

CD  There  are  more  important  work  demands 
on  my  time 

CD  |  have  limited  time  to  use  the  computer 
and  prefer  to  use  it  for  other  things 
CD  The  poll  was  no  longer  available 
CD  |  don’t  trust  your  organization  to  keep  my 
answers  confidential 
CD  |  dislike  computer  surveys 
CD  |  don’t  believe  that  survey  results  are  used 
to  improve  program/policies 
CD  other _ 

3.  Who  asked  you  to  complete  the  poll?  (Select 
only  ONE.) 

CD  CO 
CD  XO 

CD  Department  Head 
CD  Division  Head 

CD  Command  Master  Chief/Chief  of  the  Boat 

CD  Command  Career  Counselor 

CD  other _ 

CD  No  one 

4.  Are  you  serving  in  a  shore-based  or  sea-based 
command? 

CD  Shore-based  command 
CD  Sea-based  command 

5.  What  is  your  current  paygrade? 


CD 

El 

CD 

W2 

CD 

01/01 E 

CD 

E2 

CD 

W3 

CD 

02/02 E 

CD 

E3 

CD 

W4 

CD 

03/03E 

CD 

E4 

CD 

W5 

CD 

04 

CD 

E5 

CD 

05 

CD 

E6 

CD 

06 

CD 

E7 

CD 

07  and  above 

CD 

E8 

CD 

E9 

6.  How  many  carrier  groups  participated  in 
Summer  Pulse  ‘04? 


O  4 

CD  5 

©  6 
CD  / 
CD  8 


7.  Please  check  the  information  sources  below  you  normally  use 
to  get  information  about  current  operations  in  the  Navy. 

Please  select  ALL  that  apply. 

CD  GO’S  Call 

CD  Co-workers 

CD  Direct  supervisor 

CD  Morning  Quarters 

CD  Official  e-mail  /  Navy  Knowledge  Online  (www.nko.navy.mil) 

CD  Base  newspaper 

CD  Command  newsletter 

CD  Navy  Newsstand 

CD  Navy  Times 

CD  site  TV 

CD  The  internet  (e.g.,  Google) 

CD  News  channel  (i.e.,  Fox  News,  CNN,  local  news) 

CD  Newspaper  or  magazine  (i.e.  Washington  Post,  New  York 
Times,  Virginia  Pilot) 

CD  Radio 

CD  other 

CD  None 

8.  Please  rate  your  agreement  or  disagreement  with  the  following 
statements: 


a.  1  have  a  good  understanding  of  the  Fleet 

Response  Plan 

O 

O 

O 

O 

O 

b.  Implementation  of  the  Navy's  Fleet  Response 

Plan  will  improve  the  Navy's  flexibility  and 
readiness. 

O 

O 

O 

o 

O 

c.  Summer  Pulse  ‘04  adequately  demonstrated  the 
Navy’s  Fleet  Response  Plan. 

CD 

CD 

CD 

CD 

CD 

9.  Below  are  phrases  describing  people’s  behaviors.  Please  use 
the  rating  scale  to  describe  how  accurately  each  statement 
describes  you.  Describe  yourself  as  you  honestly  see  yourself 
now,  not  as  you  wish  to  be  in  the  future. 


a. 

Am  always  prepared. 

O 

O 

O 

O 

o 

b. 

Pav  attention  to  details. 

O 

O 

O 

o 

o 

c. 

Waste  mv  time. 

O 

O 

O 

o 

o 

d. 

Get  chores  done  right  awav. 

O 

O 

O 

o 

o 

e. 

Find  it  difficult  to  get  down  to  work. 

O 

O 

O 

o 

o 

f. 

Do  iust  enough  work  to  get  bv. 

O 

O 

O 

o 

o 

q. 

Don’t  see  things  through. 

O 

O 

O 

o 

o 

h. 

Carry  out  mv  plans. 

O 

O 

O 

o 

o 

i.  Make  plans  and  stick  to  them. 

O 

O 

O 

o 

o 

i.  Shirk  mv  duties. 

CD 

CD 

CD 

o 

o 

10.  Do  you  have  any  suggestions  to  increase  participation  by 
Sailors  like  you  in  these  surveys? 
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THANK  YOU 


Distribution 


AIR  UNIVERSITY  LIBRARY 

ARMY  MANAGEMENT  STAFF  COLLEGE  LIBRARY 

ARMY  RESEARCH  INSTITUTE  LIBRARY 

ARMY  WAR  COLLEGE  LIBRARY 

CENTER  FOR  NAVAL  ANALYSES  LIBRARY 

HUMAN  RESOURCES  DIRECTORATE  TECHNICAL  LIBRARY 

JOINT  FORCES  STAFF  COLLEGE  LIBRARY 

MARINE  CORPS  UNIVERSITY  LIBRARIES 

NATIONAL  DEFENSE  UNIVERSITY  LIBRARY 

NAVAL  HEALTH  RESEARCH  CENTER  WILKINS  BIOMEDICAL  LIBRARY 
NAVAL  POSTGRADUATE  SCHOOL  DUDLEY  KNOX  LIBRARY 
NAVAL  RESEARCH  LABORATORY  RUTH  HOOKER  RESEARCH  LIBRARY 
NAVAL  WAR  COLLEGE  LIBRARY 

NAVY  PERSONNEL  RESEARCH,  STUDIES,  AND  TECHNOLOGY  SPISHOCK 
LIBRARY  (3) 

PENTAGON  LIBRARY 

USAF  ACADEMY  LIBRARY 

US  COAST  GUARD  ACADEMY  LIBRARY 

US  MERCHANT  MARINE  ACADEMY  BLAND  LIBRARY 

US  MILITARY  ACADEMY  AT  WEST  POINT  LIBRARY 

US  NAVAL  ACADEMY  NIMITZ  LIBRARY 


